Rater Effects as a Function of Rater Training Context

This study examined the influence of rater training and scoring context on the manifestation of rater effects in a group of trained raters. 120 raters participated in the study and experienced one of three training/scoring contexts: (a) online training in a distributed scoring context, (b) online training in a regional scoring context, and (c) stand-up training in a regional context. After training, raters assigned scores on a four-point scale to 400 student essays. Ratings were scaled to a Rasch rating scale model, and several indices were computed for the sake of determining the degree to which individual raters manifested evidence of severity, inaccuracy, and centrality in the ratings that were assigned. The results indicate that latent trait indicators of leniency and inaccuracy are nearly perfectly correlated with raw score indicators of those rater effects. The results also reveal that the expected-residual correlation may be the most direct latent trait indictor of rater centrality. The results of this study also suggest that raters who are trained and score in online distributed environments may be less likely to exhibit centrality and inaccuracy effects.